Wearable Setup for Gesture Andmotion Based Closed Loop Audio Interaction

The wearable sensor and feedback system presented in this paper is a type of audio-haptic display which contains onboard sensors, embedded sound synthesis, external sensors, and on the feedback side a loudspeaker and several vibrating motors. The so called “embedded sonification” in this case here is an onboard IC, with implemented sound synthesis. These are adjusted directly by the user and/or controlled in realtime by the sensors, which are on the board or fixed on the human body and connected to the board via cable or radio frequency transmission. Direct audio out and tactile feedback closes the loop between the wearable board and the user. In many situations, this setup can serve as a complement to visual output, e.g. exploring data in 3D space or learning motion and gestures in dance, sports or outdoor and every-day activities. A new metaphor for interactive acoustical augmentation is introduced, the so called “audio loupe”. In this case it means the sonification of minimal movements or state changes, which can sometimes hardly be perceived visually or corporal. This are for example small jitters or deviations of predefined ideal gestures or movements. Our system is easy to use, it even allows operation without an external computer. In some examples we outline the benefits of our wearable interactive setup in highly skilled motion learning scenarios in dance and sports.